Conversation
loumir
left a comment
loumir
left a comment
There was a problem hiding this comment.
Ok for these changes . datalink represent associated data , so the data product type is event-list , the 'head' data set and associated data are in the other parts of the data link table .
|
I think I am not in phase with that. |
If the dataproduct_type of the dataset is "bundle" the single link contains everything. SO you can write its mime type directly in acces_format. Why would you like to have DataLink with several "options" in that case ? The only situation I can imagine is if you want to point towards the various subparts of the bundle using url with fragments. But that didn't seem to be the intention. |
Suppose I want to make available both an event-list and an event-bundle (i.e., the event list plus other associated data products) via separate ObsCore records (because some users just want the event list and others want the entire bundle). The end user can differentiate the query because one has dataproduct_type = 'event-list' and the other has dataproduct_type = 'event-bundle'. However I don't want to create a single bundle file for the latter that includes all the products because doing so would require duplicating a very large event list data product, which would dramatically increase archive storage requirements. How would I do this using datalink? |
Bruno, I think you need to sign off on changes to these use cases, since we state in the text that these are event-bundle use cases and I'm not sure what is the priority is here (event-bundle use case vs. datalink use case). As far as the technical content, I'm fine although I have some minor tweaks as far as to the textual descriptions in the example datalink table (e.g., the aeff is not the "effective area of the event-list" but rather the effective area of the telescope/instrument associated with the event list). |
A datalink is per definition a bundle, is not it? the access url is towards a kind of dictionnary containing access to many files... This case is perfectly in line with our current definition of event-bundle. I really disagree that for the use case "Search for event bundles via DataLink that include Cas A for a TeV spectromorphology study", the About "the aeff is not the "effective area of the event-list" but rather the effective area of the telescope/instrument associated with the event list", I agree, of course;) Well seen |
|
On Wed, Feb 04, 2026 at 09:46:59AM -0800, Ian Nigel Evans wrote:
However I don't want to create a single bundle file for the latter
that includes all the products because doing so would require
duplicating a very large event list data product, which would
dramatically increase archive storage requirements. How would I do
this using datalink?
In terms of declaration, I'd say the bundle would be a case for
<http://www.ivoa.net/rdf/datalink/core#package> (and the individual
artefacts #calibration or #this as applicable.
In terms of implementation, you would rather certainly create the
package-s on the fly. Shameless pitch: In DaCHS this would be done
in a data function
<http://docs.g-vo.org/DaCHS/ref.html#element-datafunction> that
raises a DeliverNow exception.
|
Well that is what I had thought, but Mireille said
which I think means that I lose the distinction between dataproduct_type = 'event-list' and dataproduct_type = 'event-bundle' so my requirement for the user to be able to differentiate between the event list and the event bundle is not met. |
change in description of the effective area in the DataLink table
Ian, Bruno, partial answer. Thanks for that point. I modified accordingly the aeff row description FIELD of the table in the PR |
|
On Thu, Feb 05, 2026 at 07:04:27AM -0800, Ian Nigel Evans wrote:
Well that is what I had thought, but Mireille said
> Ok for these changes . datalink represent associated data , so the data product type is event-list , the 'head' data set and associated data are in the other parts of the data link table .
which I think means that I lose the distinction between
dataproduct_type = 'event-list' and dataproduct_type =
'event-bundle' so my requirement for the user to be able to
differentiate between the event list and the event bundle is not
met.
dataproduct_type is what you have in obscore, which is another
interesting question: Publish event-list, event-bundle, or both in
obscore. My take (subject to revision depending on what the wider
concept is): mark it event-bundle and point to your datalink
endpoint, where people can choose whether to pull the whole thing or
just the event list.
In the datalink context, there are two pieces of semantics:
(1) the semantics column, which represents the relationship of the
link to the dataset. In the scenario above, I'd make #this the event
list and #package the bundle.
(2) the content qualifier, which typically is from product-type. And
that's where you'd say (product-type)#event-list for
(datalink/core)#this and (product-type)#event-bundle for
(datalink/core)#package.
|
I see what you mean. Something which has never been used with DataLink as far as I know. IN case you use datalink access_url in Obscore the dataproduct_type, the semantics FIELD in Obscore and the content_qualifier in Obscore have to be consistent. If you have dataproduct_type = event-bundle and use datalink then the semantics=#this row in the datalink MUST have content_qualifier=event-bundle and the link should be to the complete tar or MEF file. If you only want to split the bundle and give access separately to event-list, irf, auxiliary data and not to the tar/mef file then you have the special case where you will have NO #this row in your DataLink response And currently we have no semantics term to say that the link in the row is an excerpt (a subpart) of a larger set as the bundle is and which is what has been discovered via ObsCore. It's not forbidden to write a VEP and define a new term. But I think if what you want is to have DataLink response table giving access to each specific product independantly it's much easier to have dataproduct_type = event-list and then one row in the datalink response table will have semantics= #this and content_qualifier = event-list and the other irf or auxiliary products will appear in additoional rows with appropriate semantics/content_qualifier, as in 2.1.6. It does exactly what you wanted and is consitent with the spec an the current vocabulary. |
|
Le 05/02/2026 à 17:42, msdemlei a écrit :
*msdemlei* left a comment (ivoa/HighEnergyObsCoreExt#35)
<#35 (comment)>
On Thu, Feb 05, 2026 at 07:04:27AM -0800, Ian Nigel Evans wrote:
> Well that is what I had thought, but Mireille said
>
> > Ok for these changes . datalink represent associated data , so the
data product type is event-list , the 'head' data set and associated
data are in the other parts of the data link table .
>
> which I think means that I lose the distinction between
> dataproduct_type = 'event-list' and dataproduct_type =
> 'event-bundle' so my requirement for the user to be able to
> differentiate between the event list and the event bundle is not
> met.
I agree with Markus that #package semantics may be a good solution for
the bundle in the datalink response with one caveat
dataproduct_type is what you have in obscore, which is another
interesting question: Publish event-list, event-bundle, or both in
obscore. My take (subject to revision depending on what the wider
concept is): mark it event-bundle and point to your datalink
endpoint, where people can choose whether to pull the whole thing or
just the event list.
hummm. If you do that then #this in the datalink response must also be a
bundlle.
So what would be the event-list subpart ? we miss a term for that at the
moment
SO I think event-list in obscore/dataproduct_type is the right solution
In the datalink context, there are two pieces of semantics:
(1) the semantics column, which represents the relationship of the
link to the dataset. In the scenario above, I'd make #this the event
list and #package the bundle.
(2) the content qualifier, which typically is from product-type. And
that's where you'd say (product-type)#event-list for
(datalink/core)#this and (product-type)#event-bundle for
(datalink/core)#package.
Yes this works perfectly if we have obscore/dataproduct_type = event-list.
Reminder #package definition writes
All file-like items related to #this and #this itself packaged together in a single downloadable archive.
obviously it's a larger set than #this and so is the bundle with respect
to the event-list
…
—
Reply to this email directly, view it on GitHub
<#35 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AMP5LTHDRGDXLVBY2LOTQ634KNXHNAVCNFSM6AAAAACTZLFJ4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTQNJUGYYTCMRSGI>.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
This sounds like it might work for what I need.
Except that I really want the dataproduct_type = 'event-list' to point to the event list FITS file directly using access_url and an appropriate access_format because that would allow the data to be directly downloaded by the application (e.g., for visualization). If the response is a datalink table then extra steps would be required. |
|
Hi Ian
Le 05/02/2026 à 21:27, Ian Nigel Evans a écrit :
If you only want to split the bundle and give access separately to
event-list, irf, auxiliary data and not to the tar/mef file then
you have the special case where you will have NO #this row in your
DataLink response
This sounds like it might work for what I need.
And currently we have no semantics term to say that the link in
the row is an excerpt (a subpart) of a larger set as the bundle is
and which is what has been discovered via ObsCore. It's not
forbidden to write a VEP and define a new term.
But I think if what you want is to have DataLink response table
giving access to each specific product independantly it's much
easier to have dataproduct_type = event-list and then one row in
the datalink response table will have semantics= #this and
content_qualifier = event-list and the other irf or auxiliary
products will appear in additoional rows with appropriate
semantics/content_qualifier, as in 2.1.6. It does exactly what you
wanted and is consitent with the spec a the current vocabulary.
Except that I really want the dataproduct_type = 'event-list' to point
to the event list FITS file directly using access_url and an
appropriate access_format because that would allow the data to be
directly downloaded by the application (e.g., for visualization). If
the response is a datalink table then extra steps would be required.
Ah !!! you want both a direct access to the event-list and DataLink at
the same time for the associated products ?
Then you have to set acces_format to application/fits, acces_url
directly to the event-list dataset and use the DataLink service
descriptor in the ObsCore response
To do this you add the following VOTable RESOURCE to the ObsCore table
(see
https://www.ivoa.net/documents/DataLink/20231215/REC-DataLink-1.1.html
section 4.5)
<RESOURCE type="meta" utype="adhoc:service" name="RawAndCatalogDataLinks">
<DESCRIPTION>
This datalink service gives access to the raw data for the
discovered datasets as well as to catalogues of extracted sources </DESCRIPTION> <PARAM name="standardID" datatype="char" arraysize="*" value="ivo://ivoa.net/std/DataLink#links-1.1" /> <PARAM name="accessURL" datatype="char" arraysize="*" value="http://example.com/mylinks" /> <PARAM name="contentType" datatype="char" arraysize="*" value="application/x-votable+xml;content=datalink" /> <PARAM name="exampleURL" datatype="char" arraysize="*" value="http://example.com/mylinks?ID=NGC%206946" /> <GROUP name="inputParams"> <PARAM name="ID" datatype="char" arraysize="*" value="" ref="/primaryID/"/> </GROUP> </RESOURCE>
where /primaryID /can be obs_publisher_did, obs_id, or an additional
grouping id. With some clients this will allow to preset a query to the
DataLink service with the chosen ID.
|
What I want is to have two entries in ObsCore, one with dataproduct_type = 'event-list' that points directly to the event list FITS file via access_url and a second entry for dataproduct_type = 'event-bundle' that points to the entire bundle via DataLink. But for the latter I don't want to build a bundle tar file that embeds the event list because that doubles the required archive space, I just want the DataLink to return pointers to the event-list and associated bundled files. Thanks. |
5 participants
First change is in use case 1.4 : if the exposed dataset is a bundle we don't actually need the DataLink acces mode. The access_url would benefit to be directly pointing the application/fits (whatever High energy flavor for the bundle) bundle dataset.
Second change is in use case 1.6 : I created a dummy DataLink response table consistent with this use case. Just because the pseudo-code given in this use case may be more understandable with this table. In addition in this use case I changed event-bundle to event-list because if you expose event-bundle in ObsCore you don't need DataLink (except if you use url with fragments in the DataLink table but this another story and was not implied by the use case text)
I would like to propose other changes for use case 1.3 and 1.9 but I will first create an issue to explain that because it could be more controversial